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The purpose of this paper is to show how “realist synthesis ” methodology 
(Pawson, 2002) was adapted to review a large sample of community based 
projects addressing alcohol and drug use problems. Our study drew on a 
highly varied sample of 127 projects receiving funding from a national 
non-government organisation in Australia between 2002 and 2008. Open 
and pattern coding led to the identification of 10 barrier and nine enabler 
mechanisms influencing project implementation across the sample. Eight 
case studies (four demonstrating successful implementation; four 
demonstrating less than successful implementation) were used for depth 
exploration of these mechanisms. High level theories were developed, 
from these findings, on implementation effectiveness in projects 
addressing alcohol and other drug use problems. Key Words: Realist 
Synthesis, Evidence Base for Decision Making, Enablers and Barriers, 
Drug and Alcohol Programs. 


Policy makers and project funders are increasingly impelled to ensure that 
decision making in health care is based in evidence (Giesbrecht & Haydon, 2006; Ian, 
2002; Lewis, 2005; Loxley, et ah, 2004). Since its inception in 2001 the Alcohol, 
Education and Rehabilitation (AER) Foundation has funded more than 1,000 projects 
intended to reduce the burden of harm arising from alcohol and/or inhalant misuse in 
Australia. In 2008, the AER Foundation commissioned research to identify enablers and 
barriers to implementation across a diverse range of drug and alcohol related projects. In 
doing so they aimed to inform future investment in community-based work on alcohol 
and other substance use problems and issues. 

A major challenge of this research was to identify a strategy to systematically 
interrogate and make sense of a diverse set of project records pertaining to a 
heterogeneous sample of projects. This required us to adapt a methodology that would 
enable the identification of the many contextual factors associated with successful 
implementation across a broad range of activities. We were also looking for an approach 
that would allow us to develop theoretical propositions regarding requirements for 
successful project implementation in the alcohol and other drug (AOD) field. 

The paper briefly describes realist synthesis (Pawson, 2002; Pawson, Greenhalgh, 
Harvey, & Walshey, 2004, 2005) and it applications, how we adapted a realist synthesis 
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methodology to interrogate and make sense of a large and messy dataset, and the mixed 
methods approach we utilised. Project findings are discussed in detail elsewhere 
(MacLean, Berends, Hunter, Roberts, & Mugavin, 2011). The adapted realist synthesis 
methodology enabled us to identify commonalities and differences in implementation 
success across four diverse categories of intervention, and to develop suggestions to 
support future project planning and implementation. We conclude by outlining some 
limitations of our approach and directions for future work involving realist synthesis. 

What is Realist Synthesis? 

Realist synthesis is a research methodology developed as an alternative to a 
systematic review as a means to better understand the complexities of social interventions 
(Pawson, 2002; Pawson, Greenhalgh, Harvey, & Walshe, 2004; Pawson, Greenhalgh, 
Harvey, & Walshe, 2005). It builds on the stream of program evaluation that examines 
the logic or theory underpinning any given program and uses multiple methods to test the 
program theory (Pawson et ah, 2005). Realist synthesis also tests the contexts in which 
the logic/theory is most effective. Its catchcry is to examine “what works, for whom and 
in what circumstances, in what respects and how” (Pawson et al., 2004, p. v). It is 
generally used to analyse a single mechanism which generates the effect of an 
intervention, resulting in a theoretically based explanation of how projects work. 

The realist synthesis methodology examines complex social interventions by 
identifying the important components that describe the theory underlying the social 
intervention. In order to explain the realist synthesis methodology, it is helpful to explore 
what Pawson et al. (2004) meant when describing complex social interventions. A 
complex social intervention can be described as a theory of action, guided by an 
underlying rationale of why the action will influence or create change. For example, the 
concept of incentivisation was developed to describe a range of social interventions 
targeting smoking cessation (Pawson, 2002). There are a number of actors involved in 
any intervention, who may include staff, consumers, program/service funders, the 
community or key partners. The motivations, intention and interactions of these actors 
will influence the direction or impact of an intervention. For example, if staff and key 
partners have different expectations of a program, the direction of the program may 
become unclear or have less impact. 

In line with this, complex social interventions are situated within complex social 
situations; infonned and influenced by the social and structural contexts within which 
they are enacted. For example, incentivisation interventions targeting smoking cessation 
in young people may be applied within a regional setting. 

The complex social intervention is a living, active and interactive theory. It 
contains a number of steps or processes that are connected and may influence each other. 
It may have a linear flow, or may operate in an iterative, cyclical or non-linear manner. It 
can be modified and adapted throughout the implementation process, influenced by the 
actions of people involved and other contextual considerations. Finally, complex social 
interventions evolve as stakeholders become more familiar or aware of the possibilities 
(impacts and outcomes) associated with the theory of action. 

Pawson et al. (2004) use the category of ‘context’ to describe the direct 
environment in which the intervention is being implemented. Context can describe 
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different levels of intervention, including a local level (e.g. skill gaps); a population level 
(e.g. target groups, including youth or rural); or a political level (e.g. policy 
environments). 

The category of outcomes is used to describe the connection or result of 
interaction between the steps (or processes) involved in the complex social intervention. 
This is the change that was intended (or unintended) as a result of the initial action. 

The element of realist synthesis methodology that allows us to understand how 
the complex social intervention achieves its outcome is the mechanism. Realist synthesis 
is built on the premise that it is not programs themselves which have effects, but rather 
how or why they act on individuals or groups to generate change. Pawson et al. (2004) 
use the term mechanism to describe the factor or element that connects actions, situates 
them within the relevant context, and links them to the outcome. This notion allowed us 
to compare projects which were based on different activities however appeared to share a 
broadly-defined mechanism, such as engagement of community or other agencies as 
partners in their endeavours. 

The first step of a realist synthesis review is to clarify the scope of the review 
through a rigorous interrogation of the review questions, the nature of the intervention, 
the context and circumstances in which it is used and any other factors (including policy) 
that may impact its application. It involves defining and challenging the theories 
underpinning the intervention and designing an evaluative framework through which to 
explore the intervention. 

The second step in a realist synthesis review is to search for evidence about the 
intervention, primarily through a review of published literature, and to identify a defined 
set of theories about the intervention. A further review of the literature is used to test the 
usefulness of these theories in describing the social intervention. Step three involves the 
appraisal of primary studies to examine the relevance and rigour of the research. In this 
stage, data extraction forms and checklists are developed to support a systematic 
approach to data extraction, to examine the theories underpinning social interventions. 

The next step is to synthesis the collected data to refine the theory of what works, 
for whom, how and under what circumstances. The final step is to test these theories in 
practical settings and to invite feedback from practitioners and experts on the most 
appropriate recommendations for practice arising from the outcomes of the review. 

Each step should be revisited as more information is uncovered, or as different 
understandings are developed. It is a complex and multi-layered approach that ultimately 
yields a breadth of information about what works, for whom and in what circumstances. 

In the following sections we describe where realist synthesis is useful and how we 
applied realist synthesis for this project. In brief, our approach utilised different data 
sources and data collection strategies to Pawson et al (2005), using primary data to 
inform and refine the development of theories, published literature to challenge and 
confirm these theories, and a case study approach to test these theories in practical 
settings. Further to this, we adapted the approach to examine multiple social 
interventions, and to explore common theories that may operate across these 
interventions. 
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Where is Realist Synthesis Useful? 

Outcomes of public health interventions are invariably affected by the contexts in 
which they are applied (Jackson & Waters, 2005). Unlike systematic review, which seeks 
to understand whether and how an intervention works or does not work (Jackson, nd), 
realist synthesis is designed to support the generation of theories about how contexts 
affect outcomes of various kinds of program effects. Realist synthesis is useful when 
assessing whether and how a particular program or social intervention can be transferred 
to a new situation, audience or context, and the critical factors that are required to make 
that program or intervention successful. It embraces the use of mixed methods to enhance 
the rigour and validity of the research (Sheldon, 2005). Realist synthesis is particularly 
useful when dealing with a single complex social intervention that has multiple 
components, operates across multiple sites, and involves multiple actors or agents. 

McCormack et al. (2006) used realist synthesis to explore the theories 
underpinning practice development as a strategy of introducing or managing change in 
the health sector, and to examine how it is implemented across a range of contexts, using 
a range of strategies. The purpose of their research was to provide an evidence base for 
the development of a state-wide model of practice development for the health care sector. 
McConnack et al. identified a number of mechanisms that facilitate the implementation 
of practice development processes, such as strong support from staff and management. 

Our Application of Realist Synthesis 

The AER asked the Health Services Research and Evaluation team from Turning 
Point Alcohol and Drug Centre to conduct an independent review of the projects the AER 
had funded from 2002 to 2008 (from inception to the most recent funding round), and to 
develop an evidence base regarding the barriers and enablers to successful program 
implementation in the alcohol and other drug field. This project was of great interest as it 
constituted a large scale exploration of the factors that may influence project success, and 
to ultimately and ideally develop a guide to assist services in the future in strengthening 
their program and project implementation across a diverse range of circumstances. 

Our intention in this project was to identify barriers and enablers to 
implementation through an examination of interventions related to alcohol or inhalant use 
that were undertaken in community health services across Australia. Realist synthesis 
provided a systematic approach to analysis that allowed for the contextual complexities 
and diversities associated with AER funded projects. 

The project was guided by a Steering Committee composed of two representatives 
from the AER executive, including staff with strong research backgrounds, and three 
from Turning Point. The group met throughout the duration of the study and was 
consulted on key methodological and procedural decisions. The project was reviewed 
and approved by the University of Melbourne Human Research Ethics Committee. 

Developing a Framework for Analysis 

The first steps in the project were to develop a framework for analysis. In keeping 
with Pawson et al. (2005), we utilised an iterative and reflective process that involved the 
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Steering Committee in discussions around methodological issues and interpreting 
findings at a number of stages across the project life course. For example, we worked 
with the Steering Committee to develop a rationale for the identification of projects for 
inclusion in the study sample. The Steering Committee also provided advice on the 
development of analytic categories and on presentation of research findings. 

The AER’s database of 1,000 completed projects was the source of all projects 
considered within the study. A group of 127 completed projects was identified for the 
study sample through assessing each project against agreed exclusion criteria. Initial 
exclusion criteria included projects costing less than $20,000 (because they were subject 
to different reporting requirements); capital works projects; conference attendance and 
individual professional development opportunities; festivals and events; research; and 
projects related to program or service administration. 

A second round of exclusion involved assessing remaining projects against two 
criteria: relevance and rigour (Pawson et al., 2005). Programs were considered relevant 
where their intervention was similar enough to other projects so that something could be 
learned about how they worked in comparison with others. We interpreted rigour as 
having sufficient documentation from which to assess whether project objectives were 
met, defined as the presence of either a final project report or external evaluation in the 
files. 

These criteria may have resulted in a biased sample, given that organisational 
capacity may influence rigour of reporting and documentation within projects, and may 
be associated with an increased likelihood of successful implementation. These dataset 
limitations were acknowledged from the outset, but unable to be resolved further given 
the scope of the project was limited to available, existing documentation. Once the 
sample of projects for analysis was identified, our first tasks involved discussions to 
clarify the scope of the research, a review of available documentation on projects, and the 
establishment of a database to extract and analyse data. 

Appraise Primary Studies and Extract Data 

Our initial task was to identify the categories of intervention in a range of 
implementation contexts. The next step was to conduct a comprehensive review of all 
project documentation associated with each of the projects. Project documentation 
included project contracts, interim or routine reports, final reports and evaluations 
conducted by external agencies. The raw data were inputted into a database with 
accompanying coded data. The coding system for this stage of the project was developed 
with reference to the literature review, using thematic analysis, and allowing for 
flexibility in recoding and developing new codes as necessary (Hansen, 2006). This stage 
of the project involved two researchers to review a sample of project documentation and 
establish a coding schema to describe the data. 

In keeping with the strategies proposed by realist synthesis, we conducted an 
exploratory review of the literature surrounding the identified interventions. This 
literature review challenged and confirmed our analytical framework and provided 
further understanding of the different mechanisms that may influence project 
implementation. While analytic categories were developed and refined during data 
collection and analysis, the following information was gathered for each project: 
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• The context (C) of each project (including funding amount, time 
frame, urban, rural or remote location, populations targeted) 

• Project mechanism (M) (identified through analysis of enablers and 
barriers) 

• Outcomes (O) (whether projects met, partially met, did not meet or 
failed to provide evidence of meeting agreed objectives) 

We identified four categories of intervention: a), organisational enhancement, b). 
sector training, c). community education and prevention, and d). engagement and 
treatment services. Each project was identified as belonging to one of these categories of 
intervention. Table 2 provides a definition of each of these categories. 

Table 2. Categories of Intervention 


Intervention type 

Definition 

Enhancing organisational 
systems and processes 

Projects which worked to improve organisational responses to AOD 
through enhanced systems and processes 

AOD training and 
workforce development 

Projects which aimed to improve AOD service delivery or awareness 
through provision of training to the AOD workforce or to other service 
providers, for instance teachers or pharmacists 

Community education and 
prevention 

Projects that attempted to raise AOD awareness, prevent misuse or 
effect policy change by influencing the population or a large group 
within the population i.e. Indigenous people, or through local 
community development and planning 

Engagement and treatment 

Projects designed primarily to engage and influence individuals and 
groups who misuse AOD or who are at risk of doing so, or to provide 
treatment and aftercare 


Identifying what works, in what respects and how?. We were concerned to 
identify different mechanisms within each intervention, rather than focusing solely on the 
range of theories driving each social intervention. Project mechanisms were developed 
through a complex process involving two levels of analysis. This occurred through the 
data collection phase as an iterative process involving reading literature that categorised 
factors and through drawing on categories in the data. 

Given the nature of the study, a predominantly qualitative approach was required 
to identify the common and differentiating features influencing project success. It was not 
feasible to develop an understanding of project mechanisms prior to analysis of the data. 
For this reason an open coding system was used to record enablers and barriers as data 
collection proceeded (Hansen, 2006; Saldana, 2009). Broad program mechanisms could 
not be identified until all projects had been considered and an initial data analysis stage 
had been completed to identify barrier and enabling factors. 

First cycle coding entailed the development of provisional and descriptive codes 
to categorise descriptions of enablers and barriers to successful project implementation 
recorded in project files (Saldana, 2009). Barriers and enabling factors were clearly 
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identified for some projects; in others the researchers were required to make interpretive 
decisions about what constituted barriers and enabling factors from available 
documentation. Enablers and barriers recorded for only one project were excluded prior 
to further analysis. Fifty-one enabling factors and 54 barrier factors were identified. 

Second cycle coding entailed pattern coding to identify broad mechanisms that 
operated across a number of projects. Pattern codes are used to “identify an emerging 
theme, configuration or explanation. They pull together a lot of material into a more 
meaningful and parsimonious unit of analysis” (Miles & Huberman, 1994, p. 69). 
Enabling factors were grouped as nine key pattern codes and barrier factors were grouped 
as ten key pattern codes. For consistency with realist synthesis tenninology these pattern 
codes have been tenned mechanisms. 

The number of mechanisms identified across projects through thematic analysis 
created a challenge for the project, as there were far more mechanisms in this project than 
in a typical realist synthesis (Pawson et al., 2005). This was because studies included in 
our sample were not selected on the basis of sharing an underlying theory or mechanism, 
but rather mechanisms were identified within a set sample of projects. 

Nine enabling and ten barrier mechanisms were identified through the analysis. 
These mechanisms included such things as project planning and design; research and data 
collection; funding and resourcing; staffing and leadership; organisational governance 
and capacity; staff team communication and relationships; external communication and 
relationships; sensitivity to service users and settings; participatory approach to service 
delivery; and, wider service system challenges. As described above, each mechanism 
encompassed a range of enabling or barrier factors, for example, the enabling mechanism 
of external communication and relationships included factors such as community respect 
and enthusiasm for the project; involvement of local elders (in Indigenous projects); 
collaborative or consultative project development; partnerships between other agencies; 
networks with agencies to access participants; alignment with political or policy events; 
political or government support; or public relations activities such as generating media 
interest. 

In keeping with the strategies proposed by realist synthesis, we conducted an 
exploratory review of the literature surrounding the identified interventions. This 
literature review challenged and confirmed our analytical framework and provided 
further understanding of the different mechanisms that may influence project 
implementation. 

Analysis of these projects was supplemented with a further, targeted, review of 
literature documenting factors that facilitated or impeded project implementation. The 
remainder of this section discusses the steps involved in our analyses, including how we 
detennined what worked, our identification of context and our assessment of factors that 
impede or facilitate project implementation. 

In order to measure the outcomes of projects (what works), we developed a 
methodology to ascertain successful project implementation; defined as meeting all 
project objectives. We decided to record this quantitatively, to enable comparison across 
projects. We used the objectives listed in project service agreements, wherever possible. 
Where objectives were generic and did not reflect actual project activities we used 
records of anticipated outputs to measure success. 
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A global score for each project reflecting successful project implementation was 
identified using these objectives. We detennined that successful implementation included 
projects in which 100% of objectives were met (n=76 projects); partially successful 
implementation included projects in which 75% to 99% of objectives were met (n=14 
projects); and less successful implementation included project in which less than 75% of 
objectives were met (n=36 projects). 

These three categories were collapsed into two for further analysis (projects 
where 100% of objectives were met and projects where less than 100% of objectives 
were met). Although this measure effectively provided indicative results for our 
purposes, it is a subjective measure impacted by the quality and detail contained within 
project documentation. While our sample was restricted to projects that included 
evaluations, the quality and focus of these evaluations varied considerably. For a 
discussion of project findings, see Maclean et al. (2011). 

Identifying what works for whom and in what circumstances?. Project 
documentation provided us with information about the contexts that the social 
interventions were implemented in, including project location according to postcode and 
project target group (workers, Indigenous people, culturally and linguistically diverse 
communities, youth, inhalant users, communities experiencing socio-economic 
disadvantage, forensic service users, older people and alcohol users). 

Data were subsequently analysed in relation to five project contexts, which were 
selected because they were considered likely to be of interest to alcohol and other drug 
professionals, policy makers and others designing and delivering interventions. The first 
three were identified by the target populations they were designed to influence, including 
young people; Indigenous peoples; and inhalant users. These groups in particular were 
selected because the AER had prioritised funding to these groups from 2002-2008. The 
second two contexts were defined by geographic location, and include projects based in 
agencies that were based in capital cities; or based outside capital cities (including rural 
and remote areas). These contexts were selected due to an interest in exploring 
implementation differences by location. 

Using the Australian Standard Geographical Classification (2006) from the 
Australian Bureau of Statistics, we allocated each funded agency to its Statistical 
Division, determined by postcode. This procedure allowed us to differentiate an 
additional two mutually exclusive categories of projects; those based in capital cities and 
those based outside capital cities. 

Synthesise Evidence and Draw Conclusions 

Having quantified the data, statistical analysis was used to identify the distribution 
of enablers and barrier mechanisms by project type and contexts. Inferential statistical 
analyses were used to assess significant associations between project success (defined as 
meeting 100% of objectives) and other variables including contextual considerations, 
project types, and enabling and barrier mechanisms. This allowed us to identify key 
elements associated with project success. 

Our final stage of analysis included a unique case analysis to further explore the 
way these mechanisms operated to facilitate or impede project success. Although not 
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part of Pawson et al’s (2004; 2005) formulation, others have discussed the utility of case 
studies to explore and refine program theories developed through realist synthesis. For 
instance, Koenig (2009) and Leone (2008) use case studies to identify and explore 
theoretical models of what works for whom in what circumstances. Koenig suggests that 
evaluators might use case studies to test hypotheses and to refine or generate theory on 
how and in which contexts programs based on particular theoretical constructs may be 
seen to produce specific effects. 

This unique case analysis was followed by cross-case analysis examining themes, 
similarities and differences across cases. Stake advises that case studies should be 
selected not to be representative but because they may “provide insight into an issue or 
refinement of theory” developed for the project (Stake, 1998, p. 88). We selected two 
case studies from each project type as identified in Table 1; one exemplifying successful 
implementation and the other providing an example of a project which encountered 
difficulties in implementation. We also selected case study projects that allowed us to 
explore how a range of contexts impacted on implementation, and included at least one 
project involving each of the targeted populations and locational contexts. 

From this we identified skeletal theories underpinning successful project 
implementation within given contexts, and across contexts and intervention categories. 
For example, we identified that the majority of projects (78%, n=99) that achieved 100% 
of objectives noted community and/or elder enthusiasm (in the case of Indigenous 
projects) for the organisation and project, while only 22% of projects that achieved less 
than 100% of their objectives received similar support. These data supported the 
generation of a theory that community support is critical to successful project 
implementation in the alcohol and other drug arena. 

The final stage of a realist synthesis approach is the dissemination of findings and 
further testing of the identified theories. We have commenced this process but expect the 
testing of theories to occur over time as we evaluate the implementation of more projects 
within the alcohol and drug sector. 


Conclusions 

Our adaptation of the methodology demonstrated its flexibility and usefulness in 
identifying “what works, for whom and in what circumstances, in what respects and 
how” (Pawson et al., 2004, p. v). Although the findings of this project are presented 
elsewhere (Maclean et al., 2011), it is relevant here to note that the realist synthesis 
method enabled the identification of common mechanisms impacting project success 
across a diverse range of projects. Using the formulae “X works well if A, B and C are 
present” and “X does not work well if H, I and J are present”, we identified a range of 
theories operating within this intervention sample. For example, from the analysis we 
identified that engagement and treatment interventions are successful when effective 
partnerships and relationships with external agencies are present, where local networks 
are utilised and there is sensitivity to service users and treatment setting. Engagement 
and treatment interventions are less likely to be successful where the complexity of 
service users is not fully understood, where planning and design are insufficient and 
when projects are unable to find or retain staff. 
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Limitations of our Approach 

The intention of the project was to quantify mechanisms in order of the frequency 
they were observed in projects overall, and then by projects of various types and 
operating in various contexts allowing both overall conclusions to be drawn and 
comparisons between different types of projects to be made. These findings were then 
given greater depth through the qualitative case study analysis, which also enabled us to 
illustrate how mechanisms such as engaging partners in project worked (and at times did 
not work). 

Although we explored a sizeable dataset, project numbers made it difficult to 
achieve statistical significance in relation to correlations between successful 
implementation and the contextual considerations identified for analysis. Furthermore, 
when divided by intervention type the sample sets became quite small. This made it 
difficult to draw conclusions about intervention types operating within specific contexts 
(Pawson et al.’s, 2004) C+M=0). Whilst this method did provide us with a good picture 
of the most commonly observed mechanisms across interventions, we did lose some of 
the depth and detail associated with each mechanism. 

Statistical analyses, although helpful in communicating project findings, are less 
convincing for this methodology. Subjective measures were introduced to quantify 
project outcomes, sometimes with limited infonnation. Therefore the strength of the 
statistical significance identified is potentially deceptive. We suggest that caution is 
exercised when attempting to apply statistical measures in a realist synthesis analysis. 

Future Application 

Although our dataset was large in total, the numbers within each social 
intervention were quite small. In future, there would be value in exploring a larger 
dataset for each social intervention to enable greater depth exploration of a larger range 
of contextual factors influencing project success. This type of exploration would require 
more detailed project information, perhaps gained through more in-depth case study 
analyses. 

Although we applied realist synthesis across a broad range of interventions for 
comparison and commonality purposes rather than for detailed depth purposes, there is a 
richness in the data collected that could inform decision-making about project funding, 
planning and evaluation. The methodology developed for this project could be easily 
adapted to understanding factors associated with project success in other health and 
welfare areas. 
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